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I. INTRODUCTION 


A. OBJECTIVES 

The United States Navy has conducted a one-year test, 
which concluded on 1 November 1990, of the Red/Yellow/Green 
(RYG) Program. The RYG source selection improvement program 
uses contractor past quality performance as data in the source 
selection process. This program was developed to reduce the 
problems associated with poor contractor product quality. 
These poor contractor product quality problems not only impair 
fleet readiness, increase costs, and compromise safety but 
also inhibit the Government from obtaining the best purchase 
value from the contractor [Ref. 1]. 

The RYG Program is designed to use information contained 
in the Navy’s Contractor Evaluation System (CES) and the 
Product Deficiency Reporting and Evaluation Program (PDREP) 
data base to assess contractor product quality. 

This centralized data base was established at the 
direction of the Secretary of Navy to provide: 

A product deficiency reporting and data feedback system, 
maintenance of contractor/supplier quality history and 
effective use of these data to influence the pre-contract 
award process and formulate the basis for necessary post- 


award quality assurance action. [Ref. 2] 


The CES/PDREP data base is composed of the following: 


1. Contractor quality information gathered from Quality 
Deficiency Reports (QDRs) 
2. Material Inspection Record (MIR) 
3. Reports of Discrepancy (RODs) 
4. Navy Vendor Data Analysis Report (VDAR) 
5. Pre-award Surveys 
6. Defense Logistics Agency (DLA) Contractor 
Improvement Program (CIP) Alert List 
7. Method C, D, and E Corrective Action Listing 
8. Defense Contractor Management Command (DCMC) Quality 
Systems Reviews 
9. Product-Oriented Surveys 
10. Small Business Administration (SBA) Certificates of 
Competency 
These reports are forwarded to Navy Material Quality 
Assessment Office (NMQAO) via their respective Chain of 
Command. NMQAO then evaluates the reports to determine 
contractor liability and adds the appropriate entries to the 
CES/PDREP data base. Based on the information contained in 
the CES/PDREP data base RYG classifies each contractor as Red 
(high risk), Yellow (moderate risk) and Green (low risk) 
performers. Contractors who do not meet established criteria 
for RYG classifications are listed in the "Insufficient Data" 


category. Classification is done by Federal Supply 


Classification (FSC) so a contractor who produces material in 
more than one FSC may have more than one RYG classification. 
[Ref. 2] 

The RYG concept combines CES/PDREP contractor quality 
history with prescribed procedures to find the best value. RYG 
emphasizes contractor quality history by adding the cost of 
receiving poor quality goods or services into the procurement 
source selection process. One method of accomplishing this 
costing procedure is through the use of a Technical Evaluation 
Adjustment (TEA) which estimates the expected cost to the 
Government to correct or take appropriate action due to 
unsatisfactory contractor performance. 

Another method used with the Fixed Price-Greatest Value 
method of procurement is to rate the offerors in terms of 
expected quality of performance. [Ref. 2] All proposals, 
including necessary TEAS, are evaluated to determine a source 
selection that will result in the best overall contract for 
the Government. 

The focus of this research is to evaluate the 
effectiveness of the RYG Program during the initial test 
period. This research will include evaluation of actual 
contract award data during the test period, evaluation of the 
three purchase procedures classified under the RYG Program, 


evaluation of the cost avoidance/benefit and the actual 


product quality received from contracts awarded under the RYG 
test. These data reflect test results conducted at five Navy 
field activities (Naval Air Engineering Center Lakehurst, 
Naval Avionics Center Indianapolis In, Naval Ships Parts 
Control Center Mechanicsburg Pa (Code 021, Level 1/SS), Naval 
Supply Center Charleston/Naval Shipyard Charleston, and Naval 
Supply Center Pensacola/Naval Aviation Depot Pensacola) which 


was compiled by NMQAO. [Ref. 3] 


Se RESEARCH QUESTIONS 

The primary research question is: During the initial test 
period, did the RYG Program provide for improved product 
quality and/or cost avoidance, as designed? 

The research will evolve around the actual test data in 
determining the success of this program. 

The secondary research questions in this area are: 
(1) What are the procedures used for testing the categories 
(small purchase, large purchase, and Fixed Price/Greatest 
Value) in the RYG Program? 
(2) Of the contracts awarded during the test period, what were 


the benefits of the RYG Program? 


c. RESEARCH METHODOLOGY 
Initial research was conducted by reviewing and analyzing 


data collected from primary sources, including actual raw 


contract award data from each of the test sites. These data 
were collected through telephone interviews and monthly status 
reports of the RYG test to evaluate the current status of the 
test and the potential for future expansion of the program. 
Furthermore interviews were conducted involving personnel from 
the Office of the Assistant Secretary of the Navy (Research, 
Development, and Acquisition) Reliability, Maintainability, 
and Quality Assurance (ASN(RDA) RM&QA), Naval Material Quality 
Assessment Office (NMQAO), Naval Supply Systems Command 
(NAVSUP), Navy Ships Parts Control Center (SPCC), and the five 


activities involved in the test. 


D. SCOPE AND LIMITATIONS 

The scope of the thesis will be limited to the evaluation 
of the effectiveness of the RYG Program from information and 
data gathered during the one-year test period. The researcher 
will evaluate the data collected at the five test sites and 
the CES/PDREP data base maintained by NMQAO. 

The research is limited to and focuses on the test period 
from 1 August 1989 to 1 November 1990. Although some test 
activities started later and are still providing data, the 
bulk of the actual test was conducted during the above 
mentioned period and, therefore, comprises the basis for this 


thesis. 


E. ORGANIZATION OF THE STUDY 
The remainder of the thesis is organized into the 
following chapters: 
) a Chapter ee, "Background, " will provide an 
understanding of the RYG Program and how it interfaces 
with CES/PDREP. 
2. Chapter III, "RYG Test Procedures under CES/PDREP," 
will discuss the RYG Program within the structure of 
small, major, and fixed price/greatest value. 
oF Chapter IV, "CES/PDREP Program Analysis," will 
evaluate the RYG Program during its test period and 
provide a benefit analysis. 
4, Chapter V, "Conclusions and Recommendations," will 
briefly describe the effect that the RYG Program has had 
on the Navy’s material procurement quality. 
Pe SUMMARY OF FINDINGS 
The findings include the evaluation of the benefits 
provided by a comprehensive evaluation of the RYG Program 
following the one-year test period to assist Navy procurement 


activities in the determination of the feasibility of further 


implementation of the RYG Program. 


II. BACKGROUND 


A. RED/YELLOW/GREEN PROGRAM 

The Navy, like other Department of Defense components, 
continues to experience problems with contractor product 
quality. These problems impair fleet readiness, increase 
costs, and compromise safety. A key to improving quality is 
to use contractor product quality history in the contract 
award process to ensure the Navy receives the quality it 
requires. [Ref. 4] 

The Navy developed the Red/Yellow/Green (RYG) Program to 
meet the requirements of the Department of Defense (DoD) and 
Secretary of the Navy (SECNAV) policies. These policies state 
that contractor quality history will be collected and 
maintained in a centralized data base to assure that contracts 
are not awarded to contractors with a previous history of 
providing unsatisfactory quality products without determining 
required quality assurance action prior to and after contract 
award. 

Red/Yellow/Green Program is the title given to the 
methodology of evaluating and categorizing contractor quality 
performance data by Federal Supply Classification (FSC) and 


using these data to assist in the source selection process. 


Under the RYG Program, a contractor’s past quality performance 
is evaluated and assigned a color classification based upon 
the degree of risk to the Government of receiving poor quality 
products. The RYG Program does not classify contractors, but 
rather it classifies the contractor’s quality performance by 
FSC, so a contractor who provides material in more than one 
ESC may have more than one RYG classification. 

The RYG Program color classifications are: RED - High 
quality risk, YELLOW - Moderate risk, and GREEN - Low risk. 
Contractors for which there is insufficient data are assigned 
an "Insufficient Data" status. The general description of 
each color classification as outlined in the program are: 

1. RED: The performance history of the contractor for a 
given commodity indicates that he has supplied goods or 
services of poor enough quality to require the application of 
special quality assurance actions. The seriousness of the 
contractor’s negative quality history is sufficient to require 
review and approval by the head of the contracting office (as 
defined by the Federal Acquisition Regulation (FAR)) prior to 
contract award. The contractor is designated as a high 
quality risk. 

The red classification will not be utilized to bar a 
contractor from competing. The intent 18s to deter awards from 


continually poor performers and ensure that sufficient 


oversight is in place in the event that a red contractor 
receives an award. [Ref. 5] 

2. YELLOW: The performance history aindicates' the 
contractor has supplied goods or services of a particular 
commodity of poor enough quality to require special quality 
assurance actions in an effort to reduce the risk of delivery 
of poor quality products to the Navy. The contractor is 
designated as a moderate quality risk. [Ref. 5] 

3. GREEN: The performance history indicates that the 
contractor has supplied goods or services which meet or exceed 
the quality requirements of the contract. His proposal is to 
be evaluated in accordance with established acquisition 
regulations without anticipating special quality actions. The 
contractor is designated as a low quality risk. [Ref. 5] 

The specific criteria used to classify a contractor as 
Red, Yellow, or Green are listed in Appendix A. 

4. INSUFFICIENT DATA: A contractor is identified as 
having "Insufficient Data" to meet the RYG classification on 
a particular commodity if: (a) The contractor is a first time 
offeror for that FSC, (b) no quality history is available on 
the contractor for that FSC, (c) the only available quality 
information data is beyond the evaluation periods set forth in 
APPENDIX A. In the case of a contractor being classified as 


having "Insufficient Data", the Contracting Officer may elect 


to employ additional quality assurance actions. Technical 
Evaluation Adjustments (TEAS) will not be added to the 
contractor’s price during the pre-award evaluation process. 
(Ref. 5] 

It 1S important to understand that procedures set for the 
RYG Program are not designed to eliminate the requirement that 
a determination of responsibility be made for every 
prospective contractor prior to # £award. The color 
classification of a contractor alone is insufficient to 
determine responsibility of the contractor. Responsibility 
determination must be made in accordance with Federal 
Acquisition Regulation 9.104 without consideration of the 
contractor’s color classification. 

The solicitation documents and synopsis in the Commerce 
Business Daily for procurement that will be made under the RYG 
Program during the test period are required to advise 
contractors of RYG procedures and will indicate that final 
contract award will be based upon a combination of factors, 


including price and historical quality performance. 


B. CONTRACTOR EVALUATION SYSTEM/PRODUCT DEFICIENCY REPORTING 
AND EVALUATION PROGRAM 
The RYG Program uses information contained in the Navy’s 


Contractor Evaluation System (CES) and the Product Deficiency 


Ie) 


Reporting and Evaluation Program (PDREP) data base. The 
CES/PDREP are managed by the Naval Sea Systems Command 
detachment, Naval Material Quality Assessment Office (NMQAO), 
under the direction of the Office of the Assistant Secretary 
of the Navy (Research, Development, and Acquisition) 
Reliability, Maintainability, and Quality Assurance 
(ASN (RDA) RM&QA) . 

The data base iS composed of contractor quality 


information gathered from the following sources: 


1. Quality Deficiency Reports (QDRs). QDRs are 
prepared by Navy field activities to document product 
quality deficiencies, design deficiencies, or 


inadequate procurement documents resulting ag) 
defective new and newly reworked material being 
delivered to the Navy. [Ref. 5] All QDRs are 
submitted to the Naval Air Systems Command (NAVAIR), 
the Navy focal point for QDRs. Once each week, QDRs 
determined to be contractor liable and with defects 
verified, are transmitted by NAVAIR to NMQAO for 
inclusion in the PDREP data base. [Ref. 6] 


2. Material Inspection Record (MIR). MIRS are 
prepared either by Navy representatives performing 
technical inspections at a contractor’s plant or by 
Navy field activities performing technical inspections 
upon receipt of material. MIRs are submitted to the 
Navy Systems Command having cognizance over the field 
activity. [Ref. 3] The Systems Commands (NAVAIR, 
NAVSUP, Naval Facilities Engineering Command, and 
Space and Naval Warfare Command) then transmit the 


MIRs to NMQAO. The MIRs generated by NAVSEA 
activities are submitted directly ee) NMQAO. 
(Ref. 6] 

3. Reports of Discrepancy (RODs). RODs are prepared 
by Navy field activities to document receipt of 
incorrect material, shortages and overages, and 


discrepancies in preservation, packing, and marking. 
RODS are submitted to Naval Supply Systems Command 


ie 


(NAVSUP). [Ref. 5] However due to lack of real value 
of the RODs to the RYG program, RODs are no longer 
being included in the classification process; however, 
they will continue to be collected for the CES/PDREP 
program. [Ref. 7] 


4. Defense Logistics Agency (DLA) Contractor 
Improvement Program (CIP) Alert List. Contractors are 
placed on the DLA alert list if DLA has placed them in 
the CIP, if they have received a negative pre-award 
survey, or if Defense Contract Management Command 
(DCMC) has recommended they be given a pre-award 
survey for a particular reason. [Ref. 8] DLA 
sends a hard copy of the list to ASN(RDA) RM&QA. ASN 
then sends a copy to NMQAO. [Ref. 6] 


5. Navy Vendor Data Analysis Report (VDAR). The VDAR 
identifies contractors who, because of past poor 
performance, should be considered carefully before 
being awarded a contract and should be monitored after 
contract award. Evaluation of performance is based on 
data from pre-award surveys; QDRs; open DLA method C, 
D, or E corrective action; and conviction or an 
investigation for malpractice or fraud. [Ref. 8] The 
VDAR is compiled by NMQAO based on past performance 
and input from Navy Systems Commands and their field 
activities. [Ref. 6] 


6. Pre-award Surveys. Pre-award surveys are conducted 
by contract administration offices when a procuring 
contracting officer needs additional information to 
determine contractor’s management, financial 
capability, and technical skill to determine whether 
he/she will be able to perform the proposed contract. 
[Ref. 9) Only those pre-award surveys 
requested by Navy activities are included in PDREP. 
The Navy activities that requested the survey submit 
a copy of the completed pre-award survey to the 
cognizant Systems Command. The Systems Commands then 
transmit copies to NMQAO. NAVSEA activities submit 
copies of surveys directly to NMQAO. [Ref. 6] 


7. Method C, D, and E Corrective Action Listing. 
Contractors are placed on the corrective action 
listing if DLA has documented deficiencies in their 
quality programs. Specifically, method C indicates 
that the contractor has a serious quality problem or 
has not corrected a deficiency documented using method 
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B (a major deficiency). The Government sends a letter 
to the firm’s top management requesting corrective 
action. Method D indicates that less severe methods 
of corrective action (1.e., A, B, and C) have failed. 
The acquisition quality assurance program is 
discontinued, and the contractor is advised that the 
Government will not accept his goods or services until 
deficiencies have been corrected. Method E is used to 
advise a prime contractor that a subcontractor has 
quality deficiencies that would justify method C or D 
corrective action in a prime contractor and to request 
that the prime take corrective action with his 
subcontractor. fRef. 10] DCMC sends a hard 
copy of the listing to ASN(RDA) RM&QA. ASN then sends 
a copy to NMQAO. [Ref. 6] 


8. Product-Oriented Surveys. Product—Oriented surveys 
are technical product inspections conducted in a 
contractor’s plant when a buying activity desires to 
perform a special test on an item. They are performed 
by DCMC when requested by the buying activity. fa 
DCMC does not have the necessary resources, the buying 
activity may perform the survey. When a Navy activity 
requests a product-oriented survey, it submits a copy 
to the appropriate System Command. The Systems 
Commands then transmit the surveys to NMQAO. NAVSEA 
activities submit copies of surveys directly to NMQAO. 
(Ref. 6] 


9. Defense Contract Management Command (DCMC) Quality 
System Reviews. Quality system reviews are performed 
by DCMC. They involve an evaluation of the 
contractor’s quality procedures and verification that 
the contractor’s quality practices conform to those 
procedures. [Ref. 10] The reviews also evaluate the 
Government’s in-plant quality assurance program. Navy 
activities receive copies of quality system reviews if 
they participate in the review with DCMC or if they 
request a copy. Copies received by Navy activities 
are submitted to the appropriate Systems Command. The 
Systems Commands then transmit the reviews to NMQAO. 
NAVSEA activities submit copies directly to NMQAO. 
{[Ref. 6] 


10. Small Business Administration Certificates of 
Competency (COC) . If a small business is determined 
to be non-responsible by a Government buying activity, 
the small business can request that the Small Business 
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Administration (SBA) determine whether the business is 
responsible. If the SBA concludes that the small 
business is responsible, it will prepare a COC to 
document that determination. The buying activity must 
then treat the small business as a responsible 
offeror. [Ref. 3] The SBA sends hard copies of COCs 
to NMQAO for inclusion in PDREP. COCs are collected 
mainly for CES/PDREP and are not included in the RYG 
classification process. However, they do provide the 
contracting officer with a more complete profile of 
the contractor. [Ref. 11] 


The CES data base excludes: 


1. Material evaluations for base application and 
local use. 


2. Contractors developing major weapon systems. 

3. Medical procurement, material, and suppliers. 

4. Subsistence procurement, material, and suppliers. 

5. Unsatisfactory material condition caused by 
improper handling after receipt, deterioration 
during local storage, or inadequate maintenance 


or operation. 


6. Transportation discrepancies caused by the 
carrier. 


7. Ammunition and explosives accidents. 


8. Nuclear weapons procurement, material, suppliers, 
or evaluations. 


9. Naval Nuclear Power Plant primary system 
procurement, material, suppliers, or evaluations. 


10. Strategic Systems Project Office procurement, 
suppliers, or material evaluations. [Ref. 3] 


NMQAO utilizes the data base to classify the contractors 


according to the RYG Program criteria and updates the RYG 


status report monthly. 
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The CES/PDREP identifies contractors whose quality history 
may reguire the use of additional pre-award or post-award 
quality assurance actions to ensure products of the required 
quality are received. However, under CES/PDREP there is no 
procedure to determine which offeror provides the best value 
to the Navy. Consequently, what makes the RYG concept unique 
is that it combines CES/PDREP contractor quality history with 
prescribed procedures to find the best value. RYG emphasizes 
contractor quality history by adding the cost of receiving 
poor quality goods or services into the procurement source 
selection process. RYG adds this cost to the offeror’s price, 
permitting the Contracting Officer to select a contractor on 


the basis of quality and cost, rather than cost alone. 


C. VALIDITY OF CES/PDREP DATA BASE 

A major concern of the RYG Program is the validity of the 
CES/PDREP data base and the effect that this possible lack of 
validity might have on contractor protests emerging from the 
RYG Program. To ensure that contractors have every 
opportunity to challenge specific classifications, NMQAO mails 
letters on a monthly basis to Red and Yellow classified 
contractors detailing the reasons for their classification, 
the effect of the classification, and the procedures required 


to challenge the classification. During the test period, a 


ibs, 


total of 5,983 letters were mailed. Surprisingly, only 461 
responses were received, and of those only 109 were 
disagreements with the classification. Those challenges 
resulted in 53 corrections to the data base and 43 
classification changes. With less than 2% of all Red and 
Yellow classified contractors responding to the classification 
letter with challenges, and less than 1% of all contractors 
notified resulted in changes to the data base, the credibility 
of the data base has been firmly established. Furthermore, by 
sending notification letters to the contractors to inform them 
of their color classification and procedures for redress, 
NMQAO has virtually eliminated the possibility that protests 
based on the accuracy of the data base will be filed. Any 
contractor who fails to take timely action to correct the data 
base will be prevented under the rules of estoppel from 
utilizing the error in the data base as a basis for protest at 
a later date. CES/PDREP is updated monthly to include all 
corrective actions resulting from challenges and new 
information processed from all field activities. The 
Contracting Officer can then access the data base and from the 
classification and code assigned to the contractor, determine 
whether a Technical Evaluation Adjustment (TEA) should be 


added to the contractor’s proposal. 
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Ds TECHNICAL EVALUATION ADJUSTMENT (TEA) 

Technical Evaluation Adjustment’s (TEA) are the 
anticipated additional costs the Government would incur for 
taking certain additional pre-award and post-award quality 
assurance actions when the contractor for that product is 
classified as "Red" or "Yellow". TEA’s are applied based on 
whether the award is considered a small purchase (< $25,000) 
or a major purchase (> $25,000). The procedures for applying 
the TEAS are as follows: 

1. For the purposes of the RYG program, simplified small 
purchase procedures were initially defined as purchases with 
a total value in excess of $2,500 but less than $25,000. When 
RYG procedures are used for simplified small purchases, the 
purchasing agent determines the offeror’s color classification 
from the data base and assigns the applicable standard TEA 
value as listed in APPENDIX C. The standard value is derived 
from the cost of additional quality assurance actions such as 
Government Source Inspection, Receipt Inspection, and Quality 
Assurance Letter of Instruction. The cost estimates of these 
quality assurance actions which are required to be performed 
are listed in APPENDIX D. The corresponding value assigned to 
each of the quality assurance action are calculated as shown 


in APPENDIX E. 
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2. For major purchases, RYG procedures require that the 
Contracting Officer determine the offeror’s Color 
classification and code from the RYG Evaluation Criteria 
listed in APPENDIX A. Utilizing the guidelines for TEA 
assignment in APPENDIX B, the Contracting Officer can 
determine which additional quality assurance requirements the 
Government will use. The additional requirements correspond 
to estimated costs listed in APPENDIX D. These costs have 
been computed from the standard costs listed in APPENDIX E. 
The total cost of the additional quality assurance 
requirements will give the Contracting Officer the required 
TEA. 

Except for actual DCMC costs, the estimated costs listed 
in APPENDIX D are provided as examples. Each activity must 
calculate its own set of TEA costs using the format in 
APPENDIX E, since the TEA costs are based on local prevailing 
test costs and labor rates. 

The TEA represents the anticipated cost to the Government 
to correct or take appropriate quality assurance action due to 
poor previous contractor performance. The application of the 
TEA raises a contractor’s proposed price. This provides the 
Contracting Officer with the ability in the source selection 
process to obtain the supplies or services at the best overall 


value to the Government. 
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After TEA’s have been computed and added to the 
contractor’s proposals, the contract is then awarded to the 
aApPPLOpriare “contractor . If the contract is subsequently 
awarded to a Green offeror, no other action is required. If, 
however, the contract is to be awarded to a Red or Yellow 
offeror, the Contracting Officer must insure that the 
appropriate clauses are added to the contract to ensure that 
additional quality assurance actions are taken during 


performance. 


E. GREATEST VALUE / BEST BUY 

Another manner in which RYG is utilized is through 
GREATEST VALUE/BEST BUY evaluation criteria, which applies 
only to negotiated competitive solicitations. During the test 
period, the test activities developed evaluation plans and 
procedures tailored to their requirements. The evaluation 
plan considers price, which is given a minimum evaluation 
weight of 40%, and the remaining percentage apportioned only 
to quality. Point scores are then assigned according to the 
contractor’s RYG classification, and the offerors are then 
ranked according to those point scores for both factors (price 


and quality). 
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EF. SUMMARY 

This chapter described the background surrounding the 
Navy’s RYG Program. It introduced and briefly described the 
RYG Program evaluation criteria, the CES/PDREP data base which 
is the basis of the RYG Program, and the TEA and Greatest 
Value process of assigning adjustments to Red or Yellow 
contractors. The next chapter will detail the three test 


procedures developed to implement the RYG Program. 
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III. RYG TEST PROCEDURES UNDER CES/PDREP 


A. INTRODUCTION 

The RYG Program seeks to expand the CES concept by 
assigning color classifications to selected Federal Supply 
Codes (FSCs), by contractors. This effort is an attempt to 
denote the recurring problems with a particular FSC on 
previous Department of the Navy (DON) contracts. Under the 
test, procedures of the RYG program were divided into three 
categories, simplified small purchases, major purchases, and 


fixed price-greatest value procedures. 


B. EVALUATION OF SMALL PURCHASE PROCEDURES 

The simplified small purchase procedures were tested at 
all five test sites during the 12-month test period. The 
procedure was applied to all oral and written quotations 
solicited during this test period that resulted in the 
purchase orders for selected FSCs with an estimated value in 
excess of S72,500. 00: [Ref. 12] However, following 
low initial responses, this dollar threshold was lowered to 
zero for all purchase orders. [Ref. 7] 

Each activity was allowed to determine whether to use the 


Simplified small purchase procedures on blanket purchase 
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agreements (BPA), imprest fund purchases, and delivery orders 
against established contracts or General Services 
Administration (GSA) contracts. [Ref. 6] These type of 
procurements used in conjunction with the RYG program required 
the activity to submit a written purchase order to document 
the purchase. However, the Director of Contracting at each of 
these sites except for SPCC could grant a waiver concerning 
the use of these procedures. At SPCC, the Director of Hull, 
Mechanical, and Electrical (HM&E) Contracting Department must 
grant all waivers. [Ref. 1] 

The RYG small purchase procedures require that purchases 
awarded be based on the current consideration of the 
contractor’s FSC color classification at the time of source 
selection. In view of this color classification requirement 
at the time of the award, some reclassification of FSCS were 
required to facilitate proper classification of contractors. 
The periodic change of the RYG data base concerning contractor 
classifications resulted in the need to consult the data base 
for each and every procurement at the time of source 
selection. This consulting of the data base was, however, a 
very time consuming practice. It required that each test 
activity add new steps to their normal acquisition procedure 
so to enable the contracting personnel a means to the proper 


classification of the contractors. Additionally, the data 
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base was only updated on a monthly basis, causing the 
activities even more apprehension about the accuracy of the 
data base. [Ref. 7] 

Each test site was allowed to select its own set of FSCs, 
geared toward its mission and purchasing authority. The test 
sites could modify the set of FSCs during the test period to 
accommodate the changes in requirements. All the needed 
modification of these FSCS was approved by the Director of 
Contracting at each site. The overall effect of this practice 
resulted in the sites providing a very large and concentrated 
data base within each of their areas of selection. This 
concentration of data base collection enabled the RYG program 
to be utilized quickly on even insufficient data contractors. 

When the required synopsis in the Commerce Business Daily 
(CBD) was provided for the solicitations of procurement of 
material or services, the synopsis explained that, while price 
would be a significant factor in the evaluation of offerors, 
other factors, including contractor’s quality history, would 
be considered in the final decision. 

The solicitation requirements, if in written form, will 
also included a clause advising the contractor of the RYG 
procedures as provided in APPENDIX F. Oral requests for 
quotation required the information in the clause to be orally 


conveyed to the contractor. [Ref. 12] Although this practice 
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resulted in some additional time allocation by the Government 
buyers, the process was highly successful. No protests or 
complaints were received from contractors due to this 
requirement or the lack of its use. [Ref. 7] 

The process for evaluating each offeror’s RYG color 
classification (Red, Yellow, Green, or Insufficient data) is 
determined from the data base. Once the classification is 
made, the offeror’s proposal is adjusted by applying the 
proper Technical Evaluation Adjustment (TEA). A Green 
offeror’s proposal is evaluated as received since past 
performance indicates that quality is not a concern in his 
case. A Yellow offeror’s proposal requires that the standard 
simplified small purchase TEA of $1,255.00 be applied to the 
quote. This adjustment is made to align this offeror’s 
proposal with other offerors in view of the poor past quality 
history exhibited by this offeror. A Red offeror’s proposal 
requires that a TEA of $2,499.00 be applied to the quote to 
compensate the Government for the costs of additional quality 
requirements necessary. In the case of a contractor with 
Insufficient data to be classified, no TEA is added to his 
cost proposal. [Ref. 1] 

Once the TEAs are assigned to the Red and Yellow 
contractor’s proposals, source selection begins. If the 


contract is awarded to other than a Green contractor, the 


24 


additional requirements detailed in APPENDIX C must be 


performed by the Government. More importantly, if a Red 
contractor 1S awarded the contract, the head of the 
contracting office must approve the award. This additional 


work would suggest that the purchasing activities are spending 
considerably more time in awarding contracts under the RYG 
Program. However, the initial time invested in the RYG 
Program process has proven to save time and effort of not only 
contracting personnel but also production personnel using this 
material. SPCC is only able to gauge its success by the 
resulting number of Quality Deficiencies Reports (QDR) being 
processed by the fleet. To date, SPCC has shown a marginal 


improvement in material acceptance by the fleet. [Ref. 7] 


C. EVALUATION OF MAJOR PURCHASE PROCEDURES WITH TEAS 

The RYG Program limited the major purchase procedure with 
TEAS to only two sites: SPCC Mechanicsburg, PA, and NSC 
Pensacola/NADEP Pensacola. The procedures cover all 
negotiated competitive solicitations for each site’s selected 
commodities when other than the simplified, small purchase 
procedures (a value greater than $2,500.00) are utilized 
during the test period. [Ref. 13] Fach site has 
selected a set of FSCs of special interest toward its mission 


and purchasing authority. And again as with simplified small 
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purchase procedures, a waiver of these procedures was obtained 
from either the Chief of the Contracting Office except for 
SPCC, which requires authority from the Director of the Hull, 
Mechanical, and Electrical Contracting Department. [Ref. 1] 

The synopsis in the Commerce Business Daily (CBD) 
indicates that while price is a significant factor, the final 
contract award will be based on other factors such as past 
quality performance and other factors detailed in the 
solicitation. [Ref. 1] Additionally the solicitation bears a 
notice to the contractors stating the following: 

This procurement is part of a test of the Navy’s 
Contractor Evaluation System, "Red/Yellow/Green" Program. 
Award will be based upon the Contracting Officers decision 
as to which offer provides the best value to the Navy-- 
price, past quality performance, and other factors 
considered. Details are provided in the provisions 
entitled "NOTICE TO PROSPECTIVE OFFERORS (Section L) and 
"ADDITIONAL EVALUATION FACTORS—--TEST OF CONTRACTOR 
EVALUATION SYSTEM (MAJOR PURCHASE PROCEDURES)" (Section 
M). [Ref. 1] 

APPENDIX G entitled "NOTICE TO PROSPECTIVE OFFERORS" and 
"ADDITIONAL EVALUATION FACTORS~-- TEST OF CONTRACTOR EVALUATION 
SYSTEM" contains the provision of both sections L and M 
respectively of a solicitation. 

The initial process of the Major purchase procedure is 

Similar to that of the simplified small purchase procedure in 
that the Contracting Officer queries the RYG data base to 


determine the color classification of the offeror. FN 


classification of Green indicating a high quality offeror will 
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result in no TEA assignment to the offeror’s proposal. A 
Yellow or Red contractor classification will require 
additional work by the Contracting Officer. In both cases the 
contractor’s quality history will be reviewed to determine the 
appropriate amount of TEA to administer to the solicitation. 
The amount will be determined based on the quality assurance 
action required based on that contractor’s history. This 
amount will be greater for the Red contractor since a greater 
amount of quality assurance action will be required due to 
this color classification. [Ref. 2] 

At this point in the procedure is where the difference 
between simplified small purchase procedures and major 
purchase procedures become apparent. In simplified small 
purchase procedures, a standardized amount ($1,255.00 and 
2,499.00) is adjusted to the solicitation of Yellow and Red 
offerors. Under major purchase procedures the TEAS are based 
on specific quality deficiencies in the contractor’s 
performance history. Therefore, the Contracting Officer must 
first evaluate the reason for the contractor’s classification 
from the RYG data base. Next, the Contracting Officer must 
decide the quality assurance actions that the Government will 
require based on a guideline provided in APPENDIXES A and B. 
[Ref. 13] By decoding these two guidelines (APPENDIXES A and 


B) the Contracting Officer can determine the appropriate TEAs 
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to apply to each solicitation. These guidelines, however, are 
only cost estimates which are provided as examples. Based on 
its own labor rates and test costs, each site must calculate 
its own TEAS to reflect the respective area’s estimated costs. 
Initially this process 1s extremely time consuming. In view 
of this burden, NMQAO has created a model for each site based 
on cost data provided by each site to alleviate this need to 
calculate TEAs for major purchases each time. This program 
allows the site’s Contracting Officer to select from a 
computer screen the proper quality assurance action required 
for that contractor. [{Ref. 7] 

Overall, the Major Purchase Procedures are not difficult 
or time consuming given the excellent assistance provided by 
NMQAO. The Contracting Officer’s at both test sites were 


Satisfied with the program and its results. 


D. EVALUATION OF MAJOR PURCHASE WITH FIXED PRICE/GREATEST 
VALUE 
The test of RYG utilizing the "Fixed Price/Greatest Value" 
method for major purchases was limited to only three sites: 
Naval Air Engineering Center, Lakehurst NJ., Naval Avionics 
Center, Indianapolis, IN., and Naval Supply Center/ Naval 
Shipyard, Charleston SC. As with major purchases with TEAs, 


the "Fixed Price/Greatest Value" procedure covers all 
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negotiated, competitive solicitations for the selected 
commodities of interest of each site when other than 
simplified small purchase procedures are used. When sealed 
bids were appropriate this procedure was also be_ used. 
Solicitations under this procedure include a clause, as 
provided in APPENDIX H, notifying the contractors of this test 
program. Again, each site was granted the discretion of 
selecting the FSCs of interest to that site, and selecting the 
source selection/evaluation procedure most advantageous to the 
requirement. [Ref. 14] 

The sites had waiver authority through their respective 
Cnet of the Contracting Office. The only real constraint for 
the purpose of the test was the requirement that price would 
constitute at a minimum 40% of the evaluated weight, with the 
remainder apportioned according to the quality assurance 
actions required by the RYG program. The source selection 
criteria did not consider any other factors (price and RYG 
quality classifications) during the test period. [Ref. 14] 

Upon receipt of a proposal, the Contracting Office reviews 
and analyzes the contractor’s commodity performance as set in 
the CES/PDREP data base to determine the color classification. 
With this information, the staff determines each contractor’s 
applicable rating/ranking assignment (exemplified in APPENDIX 


I) in accordance with the source selection plan criteria. If 
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commodities of different offerors are within the same color 
category, the Contracting Office provided a narrative 
detailing the relative differences between the contractors’ 
proposals. [Ref. 14] 

If the contract 18s awarded to a Yellow or Red contractor, 
the Contracting Office determines the additional quality 
assurance actions required to be imposed on the contractor. 
The Contracting Officer then includes these actions in the 
contract. [Ref. 6] 

The overall effects of this process did not increase the 
administrative processing time of the operation in any of the 
three sites. In fact the overwhelming conclusion of the three 
sites was that in the long run, given the potential problems 
of poor material receipt, the program proved to be a major 


success. [Ref. 7] 


E. SUMMARY 

This chapter briefly described the three procedures 
utilized in the RYG Program. It detailed the process being 
followed within the three procedures and introduced the usage 
of the CES data base. The following chapter will evaluate the 
cost avoidance, contract awards and benefits of the RYG 


Program. 
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IV. CES/PDREP PROGRAM ANALYSIS 


A. INTRODUCTION 

This chapter evaluates the CES/PDREP program, discusses 
the maintenance and accuracy of the CES/PDREP data base, and 
evaluates the method of computing TEAs. Raga tiOnatty, at 
evaluates the cost avoidance savings and product quality 
potential of the system and the cost avoidance savings 
realized during the RYG Program test. Finally, the RYG 
Program benefits and contract awards are assessed. 

The chapter’s discussion is based on the cost saving 
potential of the RYG Program. The RYG Program test data 


provides the basis for the cost avoidance evaluation. 


B. EVALUATION OF DATA COLLECTION/DATA BASE 

The Navy’s Red/Yellow/Green (RYG) Program uses 
contractors’ product quality evaluation history to aid 
Contracting Officers and quality assurance personnel in 
determining source selection. THe Contractors “product 
quality evaluation history aids the Contracting Officer in 
assessing the risk associated with awarding a contract to the 
COntractor. The risk, based on this quality history data 


base, is the data which the RYG Program uses to determine the 
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color classification for each Federal Supply Class (FSC) of 
every contractor listed. These data, which the Contracting 
Officer uses, remain on file for several years before being 
moved to an archive file. Archiving occurs for two reasons: 
first, to reduce these data on each active file to one year; 
and secondly, to remove files which have been inactive for 
more than one year. 

The RYG Program is able to support the source selection 
process in two ways: Technical Evaluation Adjustments (TEAs) 
and Greatest Value/Best Buy. In each of the two source 
selection processes, the contractor’s quality performance 
history 1s emphasized to properly assess the offerors. The 
past contractor quality performance history is drawn from the 
centralized data base called Contractor Evaluation System 
(CES) / Product Deficiency Reporting and Evaluation Program 
(PDREP). The collection organization, Naval Material Quality 
Assessment Office (NMQAO), in Portsmouth, New Hampshire 
manages these data base for the Navy. NMQAO then produces a 
monthly report that classifies contractors based on the data 
contained in the data base and sends these results to the RYG 
test sites. NMQAO also sends a copy of these results with an 
explanation of the RYG program to each newly assigned Red and 


Yellow contractor. Within this letter, NMQAO requests that 
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any questions or concerns related to these classifications be 
forwarded to their office. [Ref. 3] 
1. Product Deficiency Reporting and Evaluation Program 

(PDREP ) 

PDREP is a system of confirmed information. The 
system is an on-line information system which brings several 
aspects of procurement and contracting together for a 
Contracting Officer. The PDREP system is designed to save tax 
dollars, improve material quality, and encourage positive 
workmanship in contractors. 

PDREP waS initiated from the Navy’s and the 
Government’s need for an effective and meaningful method of 
stopping receipt of poor quality material. It was this need 
to coordinate within a single data base the pertinent facts of 
a contractor’s past quality history and to assist contracting 
personnel in making informed source selections, that a 
centralized computer data base was developed. The system 
eliminates duplication of product deficiency information 
systems and improves the Navy’s procurement activities by 
providing consistency in accumulated findings. The system is 
available not only to the test site activities but also to all 
Systems Commands, Procuring Contracting Officers, and all 
Quality Assurance personnel desiring this type of information. 


The system iS primarily constructed from the Quality 
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Deficiency Reports (QDRs) and Report of Discrepancies (RODs) 
prepared and forwarded to the NMQAO office from all naval 
activities. 

The PDREP program, under the provisions of the DoD 
Quality Program, has developed a standard system _ for 
collecting and using seven standard types of contractor 
quality history data in the acquisition and procurement 
process. The seven standard types are: 

a. Acquisition history 

b. Contractor plant visits 

c. Product deficiency reports (QDR and ROD) 
d. Waiver/deviation requests 

e. First article/production lot tests 

f. Special quality data 

g. Technical receipts inspection results 

The core of the system is a centralized automated data 
base which provides a better picture of a contractor’s past 
quality performance by combining these reports from different 
activities. A part of PDREP is the Contractor Evaluation 
System (CES) which is the actual data base from which RYG 
extracts its information regarding contractor performance. 

2. Contractor Evaluation System (CES) 
The CES’s RYG concept is truly an evolutionary process 


which addresses costs associated with doing business with 
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other than quality contractors. The system combines an 
automated information system with the off-line manual 
processing by contracting personnel. Other attempts at 
providing incentives to contractors through emphasis on 
quality, such as the "Blue Ribbon Program" of the Air Force, 
while successful in some ways, have been limited in that they 
apply only to certain products and require contractors to 
volunteer for the program. The RYG Program uses a data base 
that consists of information derived from the award and the 
administration of literally hundreds of thousands of DoD 
contracts. 

The CES data base, as a basis for evaluating 
contractors under the RYG Program, has proven to be extremely 
sound during its test period. While it is understood that no 
system can be implemented without some unresolved problems or 
concerns, the RYG Program has, through the use of the CES data 
base, proven to the acquisition community that quality history 
data can be collected and processed into a usable format. 
This enables Contracting Officers to make better informed 
source selection decisions and insure that the Government 
receives the quality it deserves. 

3. NMQAO Data Base Issues 
The maintenance and accuracy of the NMQAO data base is 


superb. The key measure of the accuracy of the data base is 
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the response from contractors who are classified as Red or 
Yellow. As discussed in Chapter II, following the monthly 
update of the CES data base, NMQAO mails letters of 
notification to all contractors who are classified as Red or 
Yellow. During the test period, 5,983 of these letters were 
sent with only 109 companies disagreeing with their 
classification. [Ref. 4] Of these challenges, only 53 
corrections to the data base were required, giving the data 
base a better than 99% acceptance rate. This high acceptance 
rate is attributable to the quality control process that all 
input data are subjected to. 
a. Quality of the Data Base 

The major potential problem with the data base is 
its raw data input quality and accuracy because it depends on 
the individual RODs, QDRs, and other similar documents 
prepared by numerous Government activities. This 
administrative process, which can result in errors, remains 
the single most important concern of all Contracting Officers. 
The following 1s an example of this kind of error. 

In this case, a telephonic authorization from a 
Contracting Officer at Charleston Naval Supply Center granted 
a contractor permission to substitute material in filling a 
purchase order for the Charleston Naval Shipyard. When the 


shipyard received the material, and discovered that the 
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material was a substitute, a quality deficiency report was 
generated. This subsequently became an element in the CES 
data base which erroneously classified the contractor as other 
than Green. However, due to the NMQAO letter of notification, 
this error was quickly rectified. The consequence of this 
potential improper classification could have caused this 
contractor to be displaced on future contracts. This kind of 
daily administrative contract amendment could seriously impact 
the quality of the data base. 

In view of this potential problem, Contracting 
Officers anticipated that this type of error could result in 
the questioning of the equity and fairness of the RYG Program. 
This potential breach of the CES data base could formulate the 
basis for case law which could challenge the integrity of the 
RYG Program. 

To minimize problems such as the one related above, 
the raw data from field activities must pass through a two- 
step review process prior to being input into the CES data 
base. First, the field activities are required to submit CES 
input of all types through their cognizant hardware system 
command for initial review, and in the case of QDR’s and 
ROD’s, a determination of the contractor’s liability. Second, 
the input data are further reviewed by NAVAIR or NMQAO prior 


to input into the CES data base. The safety valve for the 
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system is the contractor notification letters discussed 
previously. These letters afford the contractor the 
opportunity to submit for consideration and possible 
classification correction reclamas of all input data which the 
contractor feels are erroneous. By notifying contractors of 
their classification, the reasons for that classification, and 
providing an avenue to correct any errors in the data base, 
the contractors would be estopped from challenging the data 
base’s accuracy at a later date without first responding to 
the letter of notification. 
b. Method of Computing TEA Cost 

Another concern of many Contracting Officers was 
the method of computing the dollar value of TEAs that will be 
applied to an offeror’s proposal. This concern is driven by 
the fact that not all TEA values reflect the actual costs that 
will be incurred by the contracting office for the additional 
quality assurance actions. Given that not all TEAS are based 
upon actual costs or upon local historical data, the RYG 
Program may be in violation of the laws and regulations 
concerned with integrity of the poocurenens process. 

The RYG Program utilizes fixed TEAs for simplified 
small purchases and pre-determined TEA values for major 
purchases, which are listed in APPENDIX C and D. These TEA 


values are derived from the calculations listed in APPENDIX E. 
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Given this situation, the resolution to this RYG Program 
problem is relatively simple. Contracting offices who 
participate in the RYG Program should develop their own TEAs 
based on the local prevailing labor rates and hours expended 
for each quality assurance action developed from their 
historical data. The calculations contained in APPENDIX E can 
be utilized as a guide to assist in the development of these 
local TEAs. The Contracting Officer is then at liberty to 
update the local TEAS as often as necessary to remain current 
with local market conditions. 
4. Summary 

The methods of data collection, review, and input for 
the PDREP/CES data base has proven to be very effective in 
controlling the accuracy of the data base. By providing the 
contractors with a viable notification system and procedures 
to address errors, perceived or actual, NMQAO has developed a 
sound basis for the RYG Program. Furthermore, by utilizing 
this data base, coupled with locally generated TEAS, a 
Contracting Officer is assured of a solid basis for any 
displacements which may occur during the source selection 


process. 
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C. EVALUATION OF RYG COST AVOIDANCE/PRODUCT QUALITY 

1. Evaluation of Potential Overall System Cost Avoidance 

Cost avoidance and product quality are the foundations 

of the RYG Program. These two related areas provide the 
measurement of success that the RYG Program is judged upon. 
The improvement of the overall product quality to the fleet is 
the bottom-line objective of the RYG Program. The chain 
reaction associated with the improvement of product quality 
leads not only to fewer defects in products but also to the 
subsequent reduction in cost by avoiding use of these 
defective products. The cost avoidance, which in turn is also 
gained by these improvements in product quality, can be 
measured in the following terms: 

1. Reduced corrective quality assurance action costs 

2. Rework costs 

3. Ripout costs 

4. Replacement costs 

5. "Down time costs"(most important to fleet personnel) 
The elimination of these types of costs also greatly reduces 
the risk of potential safety infractions caused by poor 
quality products. 

2. Evaluation of the Product Quality 

Product quality can not at this time be directly 


evaluated solely using the RYG Program. Cost avoidance, 
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however, can be evaluated using the RYG Program test results. 
By extrapolating the interrelationship between product quality 
and cost avoidance, the product quality evaluation as well as 
cost avoidance can be assessed. 

Based on the Navy’s RYG Program test period, it is 
estimated that over $440 million dollars a year could be saved 
by putting the program into full operation at the Navy’s 
Inventory Control Points (ICPs) (ASO and SPCC), Regional 
Contracting Centers (NRCCs), Naval Supply Centers (NSCs), 
Laboratories, and at several other large buying activities. 
A detailed summary of the figures used to calculate this 
enormous saving is provided in APPENDIX J. 

The potential cost avoidance savings due to the 
reduced incidence of repair and/or replacement of the material 
initially purchased would result in a $423 million dollar 
Savings. The cost avoidance savings resulting from not having 
to prepare and process all the QDRs and/or RODs by fleet 
activities is estimated at $7.16 million. Finally, the 
reduction of the additional quality assurance actions required 
to be performed by these same fleet activities amounts to an 
estimated cost avoidance of $16.4 million. The sum of these 
potential cost avoidance savings quickly exceeds the $440 


million dollar mark and is summarized below: 
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Repair/replacement cost avoidance ...... $422 .5,6jenS 455 


PODR. coStechuna. eee ae 1, 1S 9eZ0G. 
Additional QA actions cost avoidance ... 16,346, 840. 
SubTotal 446,067,585. 

(less cost of additional QA actions).. 6,036,594. 
Total $440,030,991. 


The calculations above are based on the following assumptions: 


1. The assumption that the test activities are supplying 
contractor quality history to the PDREP/CES data base. 


Based on discussions during the final RYG Program test 
period meeting, it was verified that all the activities who 
participated in the test program forwarded all pertinent 
contractor quality history data to NMQAO during the test 
period. Using this submission rate as a bench mark for the 
expected outcome when other field activities become a 
participant in the RYG Program, it is this researcher’s 
opinion that this assumption is_ sound. The RYG Program 
appears to motivate agency personnel. These personnel in turn 


strive to ensure that the PDREP/CES data base is up-to-date. 


2. The assumption that ten percent of an activity’s 
procurement actions would have been subject to RYG. 


As depicted in APPENDIX J, the 10% figure is not a 
scientific percentage but merely an estimate of the 


operational RYG Program actions anticipated when the listed 
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activities became RYG Program participants. This researcher 
believes this estimate is fairly reasonable based on the 
analysis provided in this section. However, there is little 
statistical data available from the test sites concerning the 
awards made within the test to make accurate predictions of 
the number of contracts or the dollar value that will actually 
apply to the RYG Program if it is fully implemented. 

In analyzing the reasonableness of the 10% estimate, this 
researcher used the Survey of Contracting Statistics (NAVSUP 
PUB 561) for fiscal years 1988/1989. An average of 852,303 
procurement actions were awarded by activities which are 
expected to participate in a fully implemented RYG Program. 
This average includes procurements for which no data are 
collected in CES and therefore are not subject to the RYG 
Program. These categories include subsistence, medical, 
nuclear weapons material and supplies, and all major weapon 
systems procurements. These excluded areas would make up 50% 
of the total procurement actions. The remaining 40% 
eliminated in the estimate could possibly be attributed to 
sole source procurements, waivers from participation, and 
continuing service contracts. The resulting /10% ‘of “total 
procurement actions which apply to the RYG Program is very 


conservative. However, when one makes estimates of possible 
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cost avoidance, to error conservatively is better than 
predicting greater savings that cannot be realized. 

The 10% estimate is also used in predicting the dollar 
value of the contract which will be competed under the RYG 
Program. In this case, the estimate 1s more acceptable 
because the procurement actions for major weapons systems, 
which are not a part of the RYG Program, account for the vast 
majority of the procurement dollar value. Again, however, the 
estimate is considered conservative and thus can be readily 
used to formulate reasonable estimates of cost avoidance. 

In conclusion, for the purpose of these cost avoidance 
calculations, the 10% estimated figure is acceptable, and does 
not represent any attempt to mislead or exaggerate the cost 
Savings potential of the RYG Program. However, if a small 
percentage of the excluded 90% of remaining contracts were 
applicable to the RYG Program, the cost savings could 
dramatically increase. Each percent change equates to a 


potential savings of $86 million. 


3. The assumption that 14 percent of the activity’s RYG 
actions would have resulted in a displacement. 


An informal survey of the activities participating in the 
RYG test Program conducted by this researcher during the RYG 
final test status meeting revealed that this figure appears to 


be high. The 14% estimate was derived by dividing the actual 
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displacements by the number of contracts awarded to Red, 
Yellow, or Green offerors (55/383 = .142). The belief is that 
a number of the contractors with an ansufficient data 
classification would not have been displaced, resulting ina 
higher number of awards to the low offeror. This fact would 
drastically reduce the $440 million cost avoidance figure, and 
as shown in APPENDIX J equates to approximately $8 million per 
percentage point difference from the 14% estimate. The effect 
of this fact could lower the estimated $440 million cost 
avoidance by $24 million with just a 3 percentage points 
differences in the calculation. This appears to be a shift 
from the conservative approach previously taken in the 
estimates of cost avoidance. To retain the conservative 
approach, the calculation of this percentage may be better 
expressed by dividing the 55 displacements by the total 
contracts awarded (including the insufficient data offerors) 
which would yield an estimated displacement factor of 
approximately 6% This would reduce the cost avoidance figure 


by $64 million dollars. 


4. The assumption that half the potential RYG displacement 
awards resulted in a Product Quality Deficiency Report 
(PQDR) because the award was to the low offeror with a 
history of providing less than requisite quality products. 
The estimated average cost of processing a PQDR is $1,200. 
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This $1,200 cost is a reasonable estimate of the total 
cost which could be avoided for the processing of QDR’s. The 
problem with this calculation stems mainly from the estimated 
displacement rate of 14%. As previously shown, the 
displacement rate may be as low as 6% which would result ina 
lower overall cost avoidance. However, the net effect of the 
6% change to the displacement rate will result in less than a 
1% change in the total realized cost avoidance of $440 
million. This would reduce the cost avoidance figure by $4 


million. 


5. The assumption that without RYG, the activities would 
have to perform additional quality assurance actions to 
reduce the risk of receiving less than requisite quality 
products from red or yellow low offerors. The average 
cost of these actions is $1,370. per award, based on the 
results of RYG test displacements. 

This assumption is based upon the concept that the 
contracting office will have to perform additional quality 
assurance actions on all contracts which would have been 
displaced by an offeror with a better quality performance 
hastomy. The RYG Program eliminates the need for these 
additional quality assurance actions when a Green offeror 
displaces a Red or Yellow offeror, and significantly reduces 
the requirement when a Yellow offeror displaces a Red offeror. 


As a result of the elimination of quality assurance 


actions through the use of the RYG Program, an estimated $16 
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million would be saved. This researcher believes that this 
figure is a good approximation of the cost avoidance savings. 
The $1,370 savings per action is based on an average of the 
cumulative total of quality actions accomplished during the 


RYG Program test. 


6. The assumption that RYG displacement awards increase 
cost by an estimated five percent. 


This assumption recognizes the fact that displacement 
will have an administrative cost (ie: personnel cost, travel 
cost etc.) to the contracting office, and to the Department of 
Defense agency accomplishing the actions required. This cost 
will be proportional to the estimated displacement action 
Savings. The estimate of 5% of the displacement cost 
avoidance is in the opinion of the researcher a reasonable 
amount, given that this includes all costs which are required 
to perform the quality assurance actions. This administrative 


cost translates to $6 million. 


7. The repair and/or replacement cost of a defective item 
is estimated to be seven times the cost of the item. [Ref. 
3] 
This assumption has the biggest impact on cost savings of 
all seven of the assumptions made by the RYG Program. The 


assumption not only accepts assumption number 3, that the RYG 


Program has a 14% displacement rate, but also assumes that 


47] 


half of the displaced awards will result in defective material 
requiring repair or replacement. In view of the use of these 
accepted assumptions, which this research has evaluated to be 
somewhat high, the $422 million contribution to the estimated 
$440 million in cost avoidance savings is suspect. This 
researcher believes that a more conservative estimate based 
on the 6% displacement factor provided in the evaluation of 
assumption 3, would equate to approximately $340 million in 
cost avoidance savings instead of $422 million. 

In conclusion, the $440 million estimate may be the 
potential cost avoidance of the RYG Program; however, the 
sensitivity analysis presented shows that the assumptions in 
the calculation are extremely dubious. As a result, the $440 
million estimate is probably too high and could be as low as 
approximately $352 million. However, the RYG Program's 
potential intrinsic value due to improved contractor quality 
may result in an even greater level of future cost avoidance. 

During an interview with Mr. W. Mackinson, Assistant 
Deputy Commander of Contracting Management, at the Naval 
Supply Systems Command (NAVSUP), the estimated $440 million 
cost savings depicted above was discussed in depth. The 
conclusion was that the evaluation presented by this 
researcher, that the $440 million was somewhat unrealistic, 


was exactly the same as his. Furthermore, reliance upon 
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figures generated primarily by statistics which are based on 
the seven asSumptions previously discussed are suspect at 
best. 
3. Red/Yellow/Green Program Test Period Cost Avoidance 

During the RYG Program Test period, a simplified 
formula for determining the savings attributable to cost 
avoidance was devised by NMQAO. Its purpose was to quantify 
the actual cost of not having to take additional quality 
assurance actions. These actions are normally taken to reduce 
the risk of receiving defective material or services from Red 
Ore xvellow contractors. The outcome was equally convincing 
that this program of assessing past contractor quality history 
was a noteworthy program. The basic formula used during the 
RYG Program test period was calculated by taking the displaced 
offeror’s price plus any TEAs less the awardee’s price plus 
any TEAs. The resulting difference was then classified as 
cost avoidance savings. The test period, up through February 


1991, produced the following total cost avoidance: 


If awarded to low offeror (with TEA) S77 9137 14s. 
Actual awards 57,799; O17 . 
Cost avoidance savings mo EG We 3) ol 


A complete summary of the test period is exhibited in APPENDIX 


Th. 
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Under the RYG Program, the required quality assurance 
action test period displacements cost avoidance has been 
determined to be approximately $1,370 per award. By taking 
the number of awards in 1989 from all major contracting 
activities (except NRCC Naples), 806,312 (APPENDIX J) and 
multiplying that number by the researcher’s estimated overall 
displacement rate of 6%, 40,000 awards could be displaced if 
the RYG Program was in effect throughout the Navy. The cost 
avoidance for these displacements would exceed six million 
dollars. This translates into a large amount of contract 
administration oversight which could be eliminated and used in 


other areas. 


D. EVALUATION OF RYG BENEFITS AND CONTRACT AWARDS 

The five test sites using the RYG Program awarded 1,014 
RYG Simplified Small Purchases and Major Purchases with TEA 
competitive procurements and 62 sole source contracts totaling 
approximately eight million dollars during the test period. 


The following table is a distribution of these procurements: 


[Ref. 4] 
NADEP/NSC Pensacola 409 
NSY/NSC Charleston 83 
SPCC Mechanicsburg 154 
NAC Indianapolis 330 
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NAEC Lakehurst O00 
Tota er 1, 076 

Of the 1,014 competitive awards, nearly two-thirds were 
classified as "Insufficient Data" awards which leaves 383 
competitive awards where the RYG Program had sufficient data 
in the PDREP/CES data base. These 383 competitive contracts 
and purchase orders awarded under the RYG Program, provided 
for 121 cases in which the low offerors had a less than 
satisfactory classification of past quality history. And of 
these 121 offerors, fifty-five were displaced. The term 
displaced means that the award was made to an offeror (other 
than the low offeror) due to the poor past quality history of 
the low offeror. The results of these figures provide the RYG 
Program and the PDREP/CES data base with a displacement rate 
of 14% (55/383 = .14). This is the figure used in the 
previously discussed assumption #3. 

Additionally, 68% of the total competitive awards made 
under the RYG Program were made to the low offeror with a 
Green classification. Of the remaining 32% of the awards, the 
RYG Program was able to, through the use of the evaluation 
criteria of the program, determine the need for all additional 
quality assurance actions to be taken by the Contracting 
Officer in order to provide a proper quality material item to 


the customer. It is the researcher’s opinion that the RYG 
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Program and the evaluation criteria provided a sound model for 
source selection. 

Furthermore, the use of contract awards based on awardee’s 
color classification requires the need for a mature and larger 
data base to generate a more accurate displacement rate. 
During the test period, 121 awards (32% of 383) made to other 
than the low offeror, 66 awards were made to the lowest 
offeror of either a Red or Yellow classification. In all 
these cases, the cost of the additional quality assurance 
actions required did not displace these offerors from 
receiving the award. However, 55 awards or 45% of the total 
awards (121) were displaced. This high displacement 
percentage quickly diminishes if the awards classified as 
"Insufficient Data" awards are added back to the calculation. 
In evaluating this relationship, a positive correlation can be 
achieved between the RYG Program displacement rate and the 
amount of quality history in the PDREP/CES data base. The 
resulting analysis of this researcher is that displacement 
rate of 14% may be artificially high due to the lack of data 
on file. The high percentage of displacements will diminish 
as the program continues. 

Another example which exemplifies the quality driving 
emphasis of the RYG Program is the fact that of the 55 


displaced awards, 26 went to contractors with Green 
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classifications for the commodity of interest and seven went 


to Yellow classified commodity holders. The remaining 
displaced awards went to "insufficient data" classified 
contractors. The researcher believes that this fact will 


induce many contractors to be more aware of their product 
quality status in the future. 

The benefits derived from the RYG Program can be measured 
in not only the expected savings derived from cost avoidance 
but more importantly from the potential increase in customer 
satisfaction, improved contractor workmanship, and better 
material quality. 

1. Customer Satisfaction. 

The end-user who receives a quality part or service is 
less stressed by the factors of "the system’s inadequacies". 
The RYG Program should not be looked at solely in terms of the 
dollars saved or costs avoided due to the reduced requirements 
for corrective quality assurance actions but rather in terms 
of increased user satisfaction due to reduced rework, 
replacement, and "down time" costs. This improved 
Satisfaction of fleet personnel is the true measure of success 
for the RYG Program or any other similar type of program. 

2. Contractor Workmanship. 
The RYG Program can be expected to enhance the 


workmanship and material quality of contractors. Contractors 
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will feel the competitive need to produce better quality 
products as this program develops and the austere budget 
picture continues. 

The final and ultimate testament of success and benefit 
resulting from the RYG Program will occur if this program 
becomes the genesis for the introduction of the Total Quality 
Management (TQM) concept within Government contractors. The 
concept revolves around the fact that product quality will 
become a measurement of a contractor, which results from an 
increase in pride and workmanship. The result could then 
easily blossom into the total quality program that Dr. Deming 
has so eloquently described in the theory of TQM. 

3. Material Quality. 

The key benefit of the RYG Program, however, is the 
reduction of down time costs by furnishing quality material to 
the fleet. The estimated $440 million in cost avoidance 
savings is largely attributable to the $422 million in savings 
from the reduction in repair/replacement costs, which in turn 
constitute the down time cost. If the RYG Program can only 
accomplish a mere 50% of that estimate, the cost savings will 
amount to $210 million. This equates to the approximate 
operating budget of the entire submarine force in fiscal year 
1990. Additionally, the morale of the repair personnel will 


improve because of the reduced need for rework and repair. 
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In this researcher’s opinion, the future benefits of 
this program are unlimited. If we are to survive the fiscal 
constraints of the future, material quality must be our 


primary concern. 


E. SUMMARY 

This chapter analyzed the results of the Navy’s RYG 
Program Test. It depicted the savings due to cost avoidance 
and the benefits of the program. It also provided a case for 
the importance of having an accurate and correct CES/PDREP 
data base from which to draw past quality history information. 
The final chapter will present the conclusions’~ and 


recommendations of the thesis. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. ANSWERS TO RESEARCH QUESTIONS 
1. During the initial test period, did the RYG Program 
provide for improved product ality and/or cost 
avoidance? 

In assessing the results of the RYG Program Test 
period, the amount of potential cost avoidance savings in the 
future was estimated at $440 million. In evaluating and 
analyzing this estimated cost saving provided by ASN (RDA) RM&QA 
this researcher determined that the estimate was somewhat 
inflated and optimistic. The potential cost avoidance savings 
from the full implementation of the RYG Program is, in the 
opinion of this researcher, approximately $300 million. This 
value is based on a more conservative estimate of the 
assumptions made by ASN(RDA) RM&QA. Additionally, the RYG 
Program Test period produced a cost savings of $113,000 
resulting from taking the displaced offeror’s price plus TEAs 
less the awardee’s price plus TEAs. Although the $113,000 
cost avoidance saving is substantially less than the estimated 
$440 million or $300 million, the fact remains that this 
figure is a significant amount of cost avoidance savings 


obtained during the test period. 
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Improvement in product quality, although a more 
illusive measurement of success, can also be extrapolated from 
the test period. The test period provided for a high 
percentage of contract awards to contractors with reliable 
performance history. This in turn improved the quality of 
material received by the Government. The RYG Program Test 
period also encouraged some contractors with poor contractor 
history to improve their performance in order to receive 
Government contracts. And finally, the RYG Program 
methodology successfully withstood the test of contractor 
PEOcest.. 

2. What are the procedures used for testing the 
categories in the RYG Program? 

The RYG Program Test period was sub-divided into three 
categories for evaluation. These categories were: 

a. Small Purchase 

b. Large Purchase 

c. Fixed Price/Greatest Value 

The procedures were discussed in detail in Chapter III 
and APPENDIX E, F, G, and H. These procedures were packaged 
and simplified for use in small purchase procedures. The 
large purchase procedures were more accommodating to the local 
factors such as labor rates and provided more flexibility to 


contracting activities. In both cases, however, Technical 
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Evaluation Adjustments (TEAS) were the means for adjusting for 
poor contractor past quality performance. The Fixed Price- 
Greatest Value procedure, though tested, did not provide any 
noteworthy results as did the previous two procedures. 
3. Of the contracts awarded during the test period, 
what were the benefits of the RYG Program? 
In evaluating the RYG Program Test period, the 
benefits received were: 
a. the cost avoidance savings of $113,000. 
b. the development of the CES data base. 
c. the consistency of the RYG Program goals with the 
goals of the DoD TQM initiative. 
d. the improved quality of material. 
e. the improved customer satisfaction and potential 
for future quality improvements associated with 


the RYG Program. 


B. CONCLUSIONS 


1. The Red/Yellow/Green (RYG) Test Program served as an 
effective method of assessing the applicability 
of the program for possible future implementation. 
The RYG Test Program proved that it could be a 
reliable tool for use in Navywide contracting activities. The 


program’s effective use of quality performance history in 
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evaluating and classifying contractors as part of the source 
selection procedures was successful. The test procedures for 
small purchases were simple to execute and easy to apply. The 
major procurement procedures, however, required more 
subjective processing by the contracting activities which, in 
turn, made the procedure more time consuming. This conclusion 
was further supported by several people interviewed during the 
RYG Program Test final meeting in Washington DC, on March 6-8, 


1991. [{Ref. 7] 


2. The RYG Program, although still in its infancy, 


is ready for Navywide implementation. 

Although participants in the RYG Program Test final 
meeting stated that the program is still in its infancy, they 
strongly recommended the implementation of this new procedure 
Navywide. Of all the people at the RYG Program Test final 
meeting, the quality assurance personnel were the most 
supportive of the program. The contracting personnel were 
more reserved in their evaluation comments due to the initial 
increase in work load, but both communities were pleased with 
the RYG Program. The quality assurance personnel seemed to be 
more convinced than the contracting personnel that the program 
would reap benefits such as reduced rework and replacement 
costs. This would far outweigh any extra work or time 


required during the initial start up process. 
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3. The award processing time and/or work load increase 
as a result of implementing the RYG Program was 
negligible. 

Contracting personnel’s two concerns were the impact 
that the RYG Program had on award processing time and work 
load. These concerns were a normal reaction resulting from 
the introduction of a new program or task. During this 
research, none of the test sites could provide any evidence of 
a significant increase in work load and/or processing time. 

4. Although the accuracy of the CES/PDREP data base 
was of initial concern to Contracting Officers, it 
was exceptionally high. 

The accuracy of the data base that the test period 
evaluated was a potential problem. The concern was repeatedly 
made in both written memorandums and at the RYG Program Test 
final meeting of 6-8 March 1991. The conclusion, however, was 
that the 99% accuracy of these data in the CES/PDREP data base 
was sustainable. Furthermore, the process employed by NMQAO 
to notify the Red and Yellow contractors of their color 
classification, following the multiple step review procedure, 
provided a very reliable means of minimizing the introduction 


of erroneous information into the data base. 
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5. The RYG Program Test period produced some 
significant cost avoidance savings to the 
Government. 

The RYG Program Test period produced some significant 
cost savings to the Government. The measure of this cost 
Savings fluctuates in part due to the nature of the process 
being evaluated. 

The RYG Program provides savings to the Government in 
several ways. First, it provides visible dollars savings to 
the customers by displacing in some instances poor contractors 
with better quality contractors. Second, the program aids the 
repair personnel by reducing the rework and replacement 
requirements through the purchase of higher quality items. 
Finally, it provides the contractor with a measurement of 


contractor quality which results in increased product quality. 


C. RECOMMENDATIONS 


[ie The Navy should expand the RYG Program to all 


contracting activities. 


The RYG Program’s ability to reduce equipment down 
time, increase quality, and improving customer satisfaction, 
while withstanding all protests initiated to date, should be 
evaluated as a measurement of success. The implementation of 


the RYG Program should phase sites in gradually. Each site 
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should be evaluated on the basis of the amount of quality 
assurance history data it has in the CES/PDREP data base. 
NMQAO should be consulted on the selection of each site anda 


Site analysis should be conducted prior to induction. 


2% The RYG Program should be used with small 


purchase procedures. 


The RYG Program is well suited for use by small 
purchase activities in source selection, contractor evaluation 
and classification due to the simple application of TEAS for 
small purchase procedures. The small purchase procedures of 
the RYG Program provide field activities with a contractor 
quality history data base, a proven evaluation method that 
accounts for poor contractor quality performance, and pre- 
packaged value for TEAS application. Furthermore, since the 
CES/PDREP data base is listed by both FSCs and contractor 
color classification, the contractor personnel can easily 
determine the contractor’s color and apply the predetermined 
TEA to that proposal. 

3. The Fixed Price/Greatest Value purchase procedures 
should not be used under the RYG Program. 

The Fixed Price/Greatest Value procedures do not adapt 
well to the RYG Program. The Fixed Price/Greatest Value 
program applies percentages to price and technical design 


instead of TEAS based on contractor past quality performance. 
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A separate program should be further evaluated and developed 
using the Fixed Price/Greatest Value methodology which could 
accommodate the complexity of this procurement program. The 
application of the RYG Program should be applied on the 
individual commodity classifications, which can be used for 
simple small purchase procedures. 

4. The CES/PDREP data base’s high degree of 


accuracy needs to be maintained. 


The task of maintaining a high quality data base is 
not only that of activities such as NMQAO, but of every fleet 
and supporting shore command in the Navy who submits quality 
information. The Navy needs to be reminded that the CES/PDREP 
data base accuracy is our responsibility. The validity of the 
RYG Program rests on the accuracy of the CES/PDREP data base. 
If error-free data are provided by all activities prior to 
being entered into the data base, a continued high quality 


data base will flourish. 


D. AREAS FOR FURTHER RESEARCH 

1. One area of further research is to evaluate and analyze 
the possibility of integration of the RYG Program with the 
existing automated procurement systems (such as APADE) which 


presently exist throughout the Navy and DoD. 
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2. Another area of research would be the development of 
a program or system that could integrate the Defense Logistics 
Agency (DLA) equivalent to CES/PDREP data base with the Navy’s 
CES/PDREP data base program. This would allow for more 
contractor past quality performance information to be analyzed 
and developed into a product with an even broader base for 


contractor classification. 
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APPENDIX A: RED/YELLOW/GREEN EVALUATION CRITERIA 


CODES AND DESCRIPTIONS 


EVALUATION CRITERION 

ON CURRENT NAVY VDAR 

METHOD C, D, AND/OR E CURRENTLY IN EFFECT 
QUALITY INFORMATION ON LATEST PRE-AWARD 
SURVEY (PAS) WITHIN LAST YEAR — NO AWARD 
LATEST PRODUCT-ORIENTED SURVEY (POS) IN LAST 
YEAR UNACCEPTABLE 

LATEST QUALITY SYSTEM REVIEW (QSR) IN LAST 
YEAR UNACCEPTABLE 

LATEST SPECIAL SURVEY IN LAST YEAR 
UNACCEPTABLE 

REJECT RATE 15% OR MORE IN LAST YEAR FOR 2 OR 
MORE LOTS 

LATEST TWO FIRST ARTICLE TESTS (FAT) IN LAST 
YEAR UNSATISFACTORY 

2 OR MORE CATEGORY "I" QDRS IN THE LAST YEAR 
6 OR MORE CATEGORY "II" ACTION QDRS IN THE 
LAST YEAR 

ON DLA CONTRACTOR ALERT LIST FOR MAJOR 


DEFICIENCIES 
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YELLOW 


RED/YELLOW/GREEN EVALUATION CRITERIA 


CODES AND DESCRIPTIONS 


ISSUED VDAR LETTER OF CONCERN 

PREVIOUSLY CLASSIFIED "RED" —- NOT WITHIN RED 
EVALUATION RANGE 

LATEST QUALITY PAS WITHIN LAST YEAR —- AWARD 
WITH FINDINGS 

LATEST POS IN LAST YEAR ACCEPTABLE WITH 
CORRECTIONS 

LATEST QSR IN LAST YEAR ACCEPTABLE WITH 
CORRECTIONS 

LATEST SPECIAL SURVEY IN LAST YEAR ACCEPTABLE 
WITH CORRECTIONS 

REJECT RATE 6-14% FOR 2 OR MORE REJECTED LOTS 
IN LAST YEAR 

LATEST FAT IN LAST YEAR UNSATISFACTORY 

ONE CATEGORY "I" QDR IN LAST YEAR 

3-5 CATEGORY "II" ACTION QDRS IN LAST YEAR 
ON DLA CONTRACTOR ALERT LIST FOR MINOR 

DER TCLENCIES 

PREVIOUSLY RED - NO REJECTS FOR 5 OR MORE 


LOTS IN LAST 6 MONTHS 
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GREEN 


RED / YELLOW/ GREEN 

CODES AND 

LATEST PAS IN 
FINDINGS 

LATEST POS IN 


LATEST QSR IN 


EVALUATION CRITERIA 


DESCRIPTIONS 


LAST YEAR -—- AWARD WITH NO 


LAST YEAR ACCEPTABLE 


LAST YEAR ACCEPTABLE 


LATEST SPECIAL SURVEY IN LAST YEAR ACCEPTABLE 


REJECT RATE LESS THAN 6% FOR 5 OR MORE LOTS 


IN LAST YEAR 


ALL FAT IN LAST YEAR SATISFACTORY 


O-2 CATEGORY 


AND G APPLIES 


wait ACTION QDRS IN LAST YEAR 


PREVIOUSLY YELLOW —- NO REJECTS FOR 5 OR MORE 


LOTS IN LAST 6 MONTHS 
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APPENDIX B: 


CODE 
A la 
B a 
C la 
D la 
E la 
EF la 
G la 
H la 
J la 
K la 
N la 
A la 
B la 
e la 
D 4, 
E 4, 
F 4, 


GUIDELINE FOR TEA ASSIGNMENT 


RED CLASSIFICATION 


ADDITIONAL QA REQUIREMENTS 


OF 


OF 


or 


or 


OF 


Or 


Or 


or 


or 


or 


OL 


or 


OB 


or 


Sao ic 


5 Or a6 


5 Ont 66 


1b, Zaeor 2D, 364 ;..) 0680 


1b, 
1b, 
1b, 
1b, 
lb, 
1b, 
1b, 
1b, 
1b, 


alley 


2a 


Or 2b, 5° OL 67) 

5 OL Of? 

oe o) ae oe | 

Or. 2b, 4, 5 or 6, 7 

5 OTF 86 71ee 

4) DOGO: 7 

Ore Zb,. 47.5 OD 6,04 

OL 2D pw oy a aor © One 
4°93 OES? 

OFwZb, 3S 4-5 -OTr (Owe 


YELLOW CLASSIFICATION 


1b, 


ILS), 


3, 
3, 


4, 


4, 


4, 


Beowro, 7 


57OLr 6. 2 


5 or 76 
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APPENDIX B 


GUIDELINE OF TEA ASSIGNMENT 





CODE ADDITIONAL QA REQUIREMENTS 
G Ao Or 16 ,;. 7 

H Zap 4," OF Or 6; 7 

9 eS 4Or 6, 7 

K Ave. OrT6,. 7 

N Zager ay 3D Or 

P Laver. lin. 4, 501 (6,7 


NOTE: The additional quality assurance actions depicted in 
this appendix are the RYG Program requirements. The 
abbreviations listed (ie: la or 2a) correspond to the quality 


assurance actions provided in APPENDIX D. 
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APPENDIX C: SMALL PURCHASE 
TECHNICAL EVALUATION ADJUSTMENTS 
RED CLASSIFICATION 
Government Source Inspection’® 
Receipt Inspection as Destination (Navy Rep)’ 


Quality Assurance Letter of Instruction”’ 


YELLOW CLASSIFICATION 
Government Source Inspection’® 


Quality Assurance Letter of Instruction’’ 


* Actual DCMC costs 
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Bo} 80) 
$1,194 


SESS 
$2,449 


$500* 


$755 
91, 2a5 


APPENDIX D: MAJOR PURCHASE 
TECHNICAL EVALUATION ADJUSTMENTS 
Quality Assurance Actions Estimated Cost 
1. Pre-Award Survey 
a. DCMC $500* 


b. PCO Representative Participation 


(1) Local’ $775 
(2) Intermediate?’ $1,380 
(3) Distant? $2,095 
2. Post-Award Orientation 
a. DCMC 22 50% 


b. PCO Representative Participation 


(1) Local‘ $1,075 
(2) Intermediate? $2,110 
(3) Distant® $3,590 


3. Product Oriented Survey (PCO Representative / DCMC) 


a. Local’ $800** 

b. Intermediate® $1,500** 

c. Distant? SZ, 2.5% 
4. Government Source Inspection’ $500* 


5. Receipt Inspection at Source (Navy and DCMC) 
a. Local” $650*** 


b. Intermediate’’ $1,360%*** 
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Coy Distant: $2,182*** 


6. Receipt Inspection at Destination (Navy) 


a. Low’ Hay 7) 
b. Medium’? $1,194 
c. High? $2, 382 
7. Quality Assurance Letter of Instruction”’ $755*** 


Notes: (1) Except for actual DCMC costs, as noted, the 
above costs are samples. Actual costS may vary between 
activities, based on each activity’s stabilized manhour rate. 

- actual DCMC cost 
wx includes actual DCMC cost — $400 


xk includes actual DCMC cost - $275 
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APPENDIX E: TECHNICAL EVALUATION ADJUSTMENT 


CALCULATIONS 


ICcalculated $30/hr x 8 hrs = $240 + $35 mileage = $275 + $500. 


27Calculated $30/hr x 8 hrs = $240 + $240 (8 hrs travel @ 
$30/hr)+ $200 (2 days per diem @ $100/day) + $200 travel costs 
= $880 + $500. 


3Calculated $30/hr x 8 hrs = $240 + $480 (16 hrs travel @ 
$30/hr) + $300 (3 days per diem @ $100/day) + $575 travel 
eosts = $1,595. 7 $500. 


‘Calculated $30/hr x 16 hrs = $480 + $45 mileage = $525 + 
$550. 


“Calculated $30/hr x 16 hrs = $480 + $480 (16 hrs travel @ 
$30/hr) + $400 (4 days per diem @ $100/day) + $300 travel 
costs = $1,660 + $550. 


SCalculated $30/hr x 16 hrs = $480 + $960 (32 hrs travel @ 
$30/hr) + $600 (6 days per diem @ $100/day) + $1,000 travel 
costs = $3,040 + $550. 


"Calculated $30/hr x 12 hrs = $360 + $40 mileage = $400 + $400 
(DCAS costs). 


°Calculated $30/hr x 12 hrs = $360 + $240 (8 hrs travel @ 
$30/hr) + $300 (3 days per diem @ $100/day) + $200 travel 
costs = $1,100 + $400 (DCAS costs). 


*Calculated $30/hr x 12 hrs = $360 + $480 (16 hrs travel @ 
$30/hr) + $400 (4 days per diem @ $100/day) + $575 travel 
costs = $1,815 + $400 (DCAS costs). 

MCalculated $34.18/hr x 14 hrs. 


“Calculated $43/hr x 8 hrs = $344 + $31 mileage = $365 + $265 
(DCAS costs). 


Calculated $43/hr x 8 hrs = $344 + $344 (8 hrs travel @ 


$43/hr) + $200 (2 days per diem @ $100/day) + $200 travel 
costs = $1,088 + $275 (DCAS costs). 


nS 


M4Ccalculated $43/hr x 8 hrs = $344 + $688 (16 hrs travel @ 
$43/hr) + $300 (3 days per diem @ $100/day) + $575 travel 
costs = $1,907 + $275 (DCAS costs). 


“Calculated $43/hr x 4 hrs = $172 + $100 material handling + 
2325 Lest Cosusr 


Calculated $43/hr x 8 hrs 
$650 test costs. 


$344 + $200 material handling + 


"Calculated $43/hr x 24 hrs = $1,032 + $500 material handling 
+ $800 test costs. 


“Calculated DCAS @ $34.18/hr x 8 hrs = $275 + $480 
(procurement representative @ $30/hr x 16 hrs). 
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APPENDIX F: CLAUSES FOR SMALL PURCHASE PROCEDURES 


NOTICE TO PROSPECTIVE OFFERORS (NOV 1988) 


(a) This procurement is subject to a test of the Navy’s 
Contractor Evaluation System (CES), "Red/Yellow/Green" (RYG) 
program. The test is authorized by the Assistant Sectary of 
the Navy (Shipbuilding and Logistics) for the acquisition of 
specific commodities within designated Federal Supply Classes 
(FSCs) by participating test activities. 


(b) The Government reserves the right to award to the 
contractor whose offer represents the best overall purchase 
value to the Government. As such, the basis for contract 
award will include an evaluation of proposed contractor’s past 
quality performance history on the particular commodity or 
commodities, identified below, as recorded in the CES. The 
price to be considered in determining best value will be the 
evaluated price after Technical Evaluation Adjustments (TEA)s 
for related quality assurance actions, as applicable, are 
applied to the offered price. 


(c) The procedures described in the clause of this 
solicitation entitled "ADDITIONAL EVALUATION FACTOR--TEST OF 
CONTRACTOR EVALUATION SYSTEM (NOV 1988)" will be used by the 
contracting officer to assist in determining the best purchase 
value for the Government--price, past quality performance, and 
other factors considered. 


(dad) The commodities included in this test, as currently 
solicited, are: 


Proc No. FSC Nomenclature CLIN 
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ADDITIONAL EVALUATION FACTORS--TEST OF CONTRACTOR EVALUATION 
SYSTEM (NOV 1988) (SIMPLIFIED SMALL PURCHASE PROCEDURES) 


(a) This procurement 18 part of a test of the Navy’s 
Contractor Evaluation System (CES) "Red/Yellow/Green" (RYG) 
Program, authorized by the Assistant Secretary of the Navy 
(Shipbuilding and Logistics), for the acquisition of specific 
commodities by participating activities. At the end of the 
test, data concerning awards made during the period will be 
evaluated to assess the program’s effectiveness and impact on 
the acquisition process. 


(b) The purpose of RYG is to assist contracting personnel 
during source selection to determine the best value for the 
Government-—-price, past quality performance, and other factor 
considered. The test program uses accumulated contractor 
quality performance on selected commodities as either "Red" 
(high risk), "Yellow" (moderate risk), "Green" (low risk), or 
"Insufficient Data", based on the degree of risk to the 
Government of receiving poor quality products. Such 
classifications are then used to apply Technical Evaluation 
Adjustments (TEA)s during source selection. 


(c) A TEA is a monetary assessment added to the price of 
selected commodities that have been classified as either 
"Red", or "Yellow" for specific contractors, and is based on 
the cost to the Government for effecting additional quality 
considerations that would otherwise not be required if award 
were made to a contractor with a satisfactory performance 
history. For purposes of requirements using the simplified 
small purchase procedures, standardized TEAs have _ been 
established for the "Red" and the"Yellow" classifications. 
During evaluation of quotations, the applicable TEA is added 
to the quoted price of the "Red" and/or "Yellow" commodity, 
and after consideration of any other pertinent price-related 
factors (e.g., transportation charges, First Article Testing, 
discount terms, etc.), becomes the basis for determining award 
of the purchase order. A commodity’s classification may 
change over time as new or revised quality performance data 
become available. 


(d) Classifications for the test program are summarized as 
follows: 


"Green"-—-Low risk. No extraordinary quality requirements or 
additional actions required; satisfactory quality history. 
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"Yellow"--Moderate risk. History of quality problems; special 
quality requirements/actions needed; Technical Evaluation 
Adjustments (TEA) applied to offered price. 


"Red"--High risk. Special alert to history of poor quality 
performance; TEA applied to offered price(s), and contract 
award requires higher level approval. 


"Insufficient Data"--Generally, may be commodities of first- 
time offerors or offerors for whom current, up-to-date quality 
performance history is unavailable; additional quality actions 
may be needed and invoked; however, a TEA is not assessed. 


(e) Prospective offerors may address questions with regard to 
their assessment classification on particular commodities to: 
Naval Sea System Command Detachment, Naval Material Quality 
Assessment Office (NMQAO), Federal Building, Room 423, 80 
Daniel Street, Portsmouth, NH 03801-3884, (Telephone) 608- 
431-9460. 
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APPENDIX G: CLAUSES FOR MAJOR PURCHASE PROCEDURES 


NOTICE TO PROSPECTIVE OFFERORS (NOV 1988) 


(a) This procurement is subject to a test of the Navy’s 
Contractor Evaluation System (CES), "Red/Yellow/Green" (RYG) 
program. The test is authorized by the Assistant Sectary of 
the Navy (Shipbuilding and Logistics) for the acquisition of 
specific commodities within designated Federal Supply Classes 
(FSCs) by participating test activities. 


(b) The Government reserves the right to award to the 
contractor whose offer represents the best overall purchase 
value to the Government. As such, the basis for contract 
award will include an evaluation of proposed contractor’s past 
quality performance history on the particular commodity or 
commodities, identified below, as recorded in the CES. The 
price to be considered in determining best value will be the 
evaluated price after Technical Evaluation Adjustments (TEA)s 
for related quality assurance actions, as applicable, are 
applied to the offered price. 


(c) The procedures described in the clause of this 
solicitation entitled "ADDITIONAL EVALUATION FACTOR--TEST OF 
CONTRACTOR EVALUATION SYSTEM (NOV 1988)" will be used by the 
contracting officer to assist in determining the best purchase 
value for the Government--price, past quality performance, and 
other factors considered. 


(d) The commodities included in this test, as currently 
solicited, are: 


FSC No. FSC Nomenclature CLIN 
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ADDITIONAL EVALUATION FACTORS--TEST OF CONTRACTOR EVALUATION 
SYSTEM (NOV 1988) (MAJOR PURCHASE PROCEDURES) 


(a) This procurement 1s part of a test of the Navy’s 
Contractor Evaluation System (CES) "Red/Yellow/Green" (RYG) 
Program, authorized by the Assistant Secretary of the Navy 
(Shipbuilding and Logistics), for the acquisition of specific 
commodities by participating activities. At the end of the 
test, data concerning awards made during the period will be 
evaluated to assess the program’s effectiveness and impact on 
the acquisition process. 


(b) The purpose of RYG is to assist contracting personnel 
during source selection to determine the best value for the 
Government--price, past quality performance, and other factor 
considered. The test program uses accumulated contractor 
quality performance on selected commodities as either "Red" 
(high risk), "Yellow" (moderate risk), "Green" (low risk), or 
"Insufficient Data", based on the degree of risk to the 
Government of receiving poor quality products. Such 
classifications are then used to apply Technical Evaluation 
Adjustments (TEA)s during source selection. 


(c) A TEA is a monetary assessment added to the price of 
selected commodities that have been classified as either 
"Red", or "Yellow" for specific contractors, and is based on 
the cost to the Government for effecting additional quality 
considerations that would otherwise not be required if award 
were made to a contractor with a satisfactory performance 
history. During evaluation of quotations, the necessity for 
any additional quality assurance requirements will be 
determined, and the applicable TEA will be assessed onto the 
quoted price of the "Red" and/or "Yellow" commodity. After 
consideration of any other pertinent price-related factors 
(e.g., transportation charges, First Article Testing, discount 
terms, etc.), this adjusted price becomes the basis for 
determining award of the purchase order. A commodity’s 
classification may change over time as new or revised quality 
performance data become available. 


(d) Classifications for the test program are summarized as 
follows: 


"Green"--Low risk. No extraordinary quality requirements or 
additional actions required; satisfactory quality history. 
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"Yellow"-—-Moderate risk. History of quality problems; special 
quality requirements/actions needed; Technical Evaluation 
Adjustments (TEA) applied to offered price. 


"Red"--High risk. Special alert to history of poor quality 
performance; TEA applied to offered price(s), and contract 
award requires higher level approval. 


"Insufficient Data"--Generally, may be commodities of first- 
time offerors or offerors for whom current, up-to-date quality 
performance history is unavailable; additional quality actions 
may be needed and invoked; however, a TEA is not assessed. 


(e) Prospective offerors may address questions with regard to 
their assessment classification on particular commodities to: 
Naval Sea System Command Detachment, Naval Material Quality 
Assessment Office (NMQAO), Federal Building, Room 423, 80 
Daniel Street, Portsmouth, NH 03801-3884, (Telephone) 608- 
431-9460. 
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APPENDIX H: CLAUSES FOR FIXED PRICE/GREATEST VALUE 
PROCEDURES 


NOTICE TO PROSPECTIVE OFFERORS (NOV 1988) 


(a) This procurement is subject to a test of the Navy’s 
Contractor Evaluation System (CES), "Red/Yellow/Green" (RYG) 
program. The test is authorized by the Assistant Sectary of 
the Navy (Shipbuilding and Logistics) for the acquisition of 
specific commodities within designated Federal Supply Classes 
(FSCs) by participating test activities. 


(b) The Government reserves the right to award to the 
contractor whose offer represents the best overall purchase 
value to the Government. As such, the basis for contract 
award will include an evaluation of proposed contractor’s past 
quality performance history on the particular commodity or 
commodities, identified below, as recorded in the CES. 


(c) The procedures described in the clause of this 
solicitation entitled "ADDITIONAL EVALUATION FACTOR--TEST OF 
CONTRACTOR EVALUATION SYSTEM (NOV 1988)" will be used by the 
contracting officer to assist in determining the best purchase 
value for the Government--price, past quality performance, and 
other factors considered. 


(d) The commodities included in this test, as currently 
solicited, are: 


FSC No. FSC Nomenclature CLIN 
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ADDITIONAL EVALUATION FACTORS--TEST OF CONTRACTOR EVALUATION 
SYSTEM (NOV 1988) (FIXED PRICE--GREATEST VALUE PROCEDURES) 


(a) This procurement is part of a test of the Navy’s 
Contractor Evaluation System (CES) "Red/Yellow/Green" (RYG) 
Program, authorized by the Assistant Secretary of the Navy 
(Shipbuilding and Logistics), for the acquisition of specific 
commodities by participating activities. At the end of the 
test, data concerning awards made during the period will be 
evaluated to assess the program’s effectiveness and impact on 
the acquisition process. 


(b) The purpose of RYG is to assist contracting personnel 
during source selection to determine the best value for the 
Government-—-price, past quality performance, and other factor 
considered. The test program uses accumulated contractor 
quality performance on selected commodities as either "Red" 
(high risk), "Yellow" (moderate risk), "Green" (low risk), or 
"Insufficient Data", based on the degree of risk to the 
Government of receiving poor quality products. A commodity’s 
classification may change over time as new or revised quality 
performance data become available. 


(c) For the purpose of source evaluation and selection, both 
the color classification of an offeror’s commodity and the 
proposed price(s) shall be evaluated in accordance with 
weighted evaluation criteria established by the Government 
prior to the receipt of proposals. Price-related factors, 
such as transportation charges, First Article Testing, 
discount terms, etc., will also be considered; however, no 
score or rating shall be applied. 


(d) Offerors are advised that, although price is of 
Significance in determining the successful offeror, past 
quality performance on the proposed commodity (as classified 
with the RYG data base) is essentially more important, and 
shall be evaluated accordingly. Each of the RYG 
classifications and its relative order of importance is 
summarized as follows: 


"Green"--Low risk. No extraordinary quality requirements or 
additional actions required; satisfactory quality history. 
Commodities within this classification are apportioned a 
greater weight or value in the evaluation than those 
classified as either "Yellow" or "Red". 
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"Yellow"--Moderate risk. History of quality problems; special 
quality requirements/actions needed. Due to the additional 
quality assurance considerations that may be necessary, 
commodities within this classification are weighted less than 
those classified as "Green", but are of greater value than 
those within the "Red" category. 


"Red"--High risk. Special alert to history of poor quality 
performance; contract award requires higher level approval. 
These commodities are apportioned the least available weight 
or value for past quality performance relative to commodities 
within the "Green" or "Yellow" classifications. 


"Insufficient Data"--Generally, may be commodities of first- 
time offerors or offerors for whom current, up-to-date quality 
performance history is unavailable; additional quality actions 
may be needed and invoked; however, commodities within this 
classification shall be evaluated solely on the basis of price 
and related factors. Past quality performance will not bea 
consideration in the evaluation of commodities for which 
current quality performance data is not set forth within the 
data base. 


(e) Prospective offerors may address questions with regard to 
their assessment classification on particular commodities to: 
Naval Sea System Command Detachment, Naval Material Quality 
Assessment Office (NMQAO), Federal Building, Room 423, 80 
Daniel Street, Portsmouth, NH 03801-3884, (Telephone) 608- 
431-9460. 
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APPENDIX I: FIXED PRICE - GREATEST VALUE 
SAMPLE EVALUATION PROCEDURES 


Source Selection/Evaluation Method 
(The following example is illustrative of a source 
selection/evaluation method incorporated by RYG test 
procedures) 


Total Points (MAX) 
Total Technical 
Total Price 


100 points (%) 
60 points (%) 
40 points (%) 


CES Classification Technical Score 
Green 60 points 
Yellow 62 pent s 
Red 10 points 
Insufficient Data 60 points 


Price Score 


Within 0 percent - 5 percent of low offeror: GREEN 
Within 5+ percent - 15 percent of low offeror: YELLOW 
Within 15 percent of low offeror: RED 
Green 40 points 

Yellow 26 points 

Red 13 points 


POSSIBLE OUTCOMES 


COLOR SCORE 
RANKING TECH PR Ler TECH PREECE TOTAL SCORE 
1 G G 60 40 100 
Zz G x 60 ZG 86 
3 x G Gis. 40 715 
4 G R 60 ies is 
5) x XY 55 Ze 61 
6 R G 10 40 50 
7 i R 32) iL: 48 
8 R NG 1,0: 26 36 
2 R R 10 iS, Zo 
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APPENDIX J: RYG COST AVOIDANCE CALCULATIONS 


Procurement Actions Fiscal Years 1988/1989 (From Survey of 


Contracting Statistics, NAVSUP Publication 561) 


1988 1989 
Actions $ (000s) Actions $ (000s) 
ICPs 107,437 2864) 250 89,896 ZF SO OS 
NRCCs (less Naples) 32,170 1, 7t7T, 039 25,159 462,077 
NSCs 420,568 1,122,162 355,977 #1,063,283 
NAVAL LABS 148,128 3678, 260 1537543 1,944,414 
Miscellaneous 

NAS CORPUS CHRISTIE 5,196 4,866 5,114 6,207 
NAC INDIANAPOLIS 22, 945 494,852 18,769 326. 7G) 
MCAS CHERRY PT. 19,634 2S 2S 137.2038 24,160 
NAS LAKEHURST LO; 07 TAA2Z76 10,687 15 99.6 
NAS PAX RIVER 19,956 262,261 Hes 119 255. .065 
NAS POINT MUGU eT SZ 3525056 16,931 49,410 
NSY NORFOLK 2357-625 697,032 9,810 32,459 
NSY PORTSMOUTH U37 312 62,013 14,980 a7 857 
NSY MARE ISLAND 16,519 40,899 205027 49,140 
NSY PEARL HARBOR 2,623 3,781 8,367 15,564 
NWC CRANE 19,700 138,408 18,640 1467500 
NOS INDIAN HEAD Ty Oo4 64,788 8,279 SSH.) 14 
NOS LOUISVILLE 10,896 63,879 LQG 258 46,675 
NOS BAY ST. LOUIS 6, 393 11,638 6,068 1372505 

TOTAL 898,293 87,749,173 B06 3h2 8,498,236 


Average for Activities during FY 88/89: 852,303 actions for 
S625, 705, 000. 


RYG DATA USING FY 88/89 FIGURES 


RYG Test Displacement Rate - 14% 
Displacement during RYG test - 55 
Competitive awards - low offeror is color classified -383 


ee —) VO 455/363) 
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Estimated RYG Actions: 85, Z30Gfores362, 3707500. 
Average FY 88/89 actions for above sites: 852,303 for 
$87,623,705, 0U0r 
Estimated percentage subject to RYG —- 10% 
85-230 = 10% of 852-303 


S8062,070, 200 = 10, otee 022,705,008 


Estimated RYG Displacement Actions: 11,932 for $120,731, 870. 
RYG Test Displacement Rate —- 14% 
Estimated RYG actions - 85,230 for $862,370,500. 
11,932 = 14% of 85,230 


$120,731,870 = 14% of 862,370,500. 


Estimated Repair/Replacement Cost: $422,561,545. 


NOTE: Since RYG is not now operational at the sample 
implementation sites, the estimated 11,932 RYG 
displacement actions above represent awards to 
red or yellow low offerors. If half of these 
awards results in defective material, the cost 
of repairing/replacing the defective material 
is estimated to be seven times the cost of the 
material. 


Estimated operational RYG action dollars - $120,731,870. 
$422,561,545. = 7 x (.5 x $120,731, 870.) 

Estimated Product Quality Deficiency Report (PQDR) Cost: 

97, 159 72e0e 

NOTE: Since RYG is not now operational at the sample 

implementation sites, the estimated 11,932 RYG 


displacement actions above represent awards to 
red or yellow low offerors. If half of these 
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awards results in defective material, Product 
Quality Deficiency Reports would be issued on 
each defective product. 


Estimated RYG displacement actions - 11, 932 
PQDR average preparation/processing cost - $1,200. 


loo, Z00 = Sie 200 & (25-2 11,932) 


Estimated additional Quality Assurance Actions Cost: 
$16,346,840. 
Estimated RYG displacement Actions - 11,932 
Additional QA actions estimated cost - $1,370/action 


9167 346,340 — 51,370 x 11, 932 
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APPENDIX K: RED/YELLOW/GREEN TEST STATUS REPORT 


Prepared 28 FEB 1991 


EXECUTIVE SUMMARY 
RED/YELLOW/GREEN PROCUREMENTS 


RYG Average 
Number Dollars ($) PALT 
Total RYG Procurements ... 1668 
Walved)):.”. -mueate . manne e 111 
AWALOGO 2 ein. oes. TS 13,205 006 50 
Ie a. Competatiane 2-o.. 1453 12,068,501 Sal 
Dw Sole. Sourceee..... 104 1713 7,305 32 
TI. a. S$252000-and vince: lots 1 32S 38k 48 
bi (OVER Z 5 000 meee 42 5, 682m4Z5 111 


Cost Comparisons 
(Competitive awards using TEAs. No Greatest Value/Best 
Buy Awards) 

1. Award to low offeror - with no TEAs: 


If all RYG procurement awards were to low 
offerors with no TEAS # Cost(S)...8)... 22 7,599,298. 


2. Award to low offeror - with TEAs: 
If all RYG procurement awards were to low 


offeror With, TEAS Costs (5). 7,913, 7452 
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Actual awards: 

The actual RYG procurement awards with TEAs 
for RED or Yellow awardee’s. Cost (S).... 7,799,917. 

Cost Avoidance: 

Cost avoidance is the low offerors price plus 


TEAS minus the awardee’s price plus TEAS (if any). 
SSR CS I ae gn me 7 EY ene eget nr er en er DS O26. 


Sg 


APACE 


ASN 


BIS 


BPA 


CAGE 


CBD 


CEDES 


CES 


CLE 


Coc 


DCMC 


DLA 


DLSIE 


DoD 


DON 


FAT 


FAR 


FMSO 


Foc 


GSA 


GSI 


APPENDIX L: GLOSSARY OF ACRONYMS 
Automation of Procurement and Accounting Data Entry 
Office of the Assistant Secretary of the Navy 
Buyer Information Service 

Blanket Purchase Agreement 

Commercial And Government Entity 

Commerce Business Daily 

Contractor Evaluation Data Entry System 
Contractor Evaluation System 

Contractor Improvement Program Alert List 
Certificate of Competency 

Defense Contractor Management Command 

Defense Logistics Agency 

Defense Logistics Studies Information Exchange 
Department of Defense 

Department of the Navy 

First Article Testing 

Federal Acquisition Regulation 

Fleet Material Support Office 

Federal Supply Classification 

General Services Administration 


Government Source Inspection 
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HM&E 
MIR 
MODEM 
NAVAIR 
NAVSEA 
NAVSUP 
NMQAO 
PAS 
PCO 
POREP 
PMRS 
POS 

QA 

QDR 
QSR 
ROD 
SBA 
SECNAV 
SPCC 
SS 
SSPO 
TEA 


VDAR 


Hull, Mechanical, and Electrical 
Material Inspection Record 

Modulator / Demodulator 

Naval Air Systems Command 

Naval Sea Systems Command 

Naval Supply Systems Command 

Navy Material Quality Assessment Office 
Pre-award Survey 

Procuring Contracting Officer 

Product Deficiency Report and Evaluation Program 
Procurement Management Reporting System 
Product—-Oriented Survey 

Quality Assurance 

Quality Deficiency Report 

Quality System Review 

Report of Discrepancy 

Small Business Administration 

secretary of the Navy 

Navy Ships Parts Control Center 
Sub-Safe/Level 1 

Strategic Systems Project Office 


Technical Evaluation Adjustment 


Navy Vendor Data Analysis Report 
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